3 CONNECTIVITY 
ancl CREATIVITY 


in times of CONFLICT 


Digital synesthesia in product design. 
Building a vocabulary of physical interactions 
for a sensible quantified self 


Viktor Malakuczi, Miriam Saviano, Sara Muscolo, 
Patrizio Cipollone, Camilla Gironi, Marco Angelini 


Sapienza University of Rome 
viktor.malakuczi@uniromat.it, miriam.saviano@uniroma_.it, sara.muscolo@uniroma_.it, 
patrizio.cipollone@uniromal.it, camilla.gironi@uniromaLit, angelini@diag.uniromal.it 


Abstract 

The paper explores the possibilities of presenting quantita- 
tive information in qualitative, expressive, engaging, and en- 
tertaining ways. To do so, experiments focus on a range of 
sporting activities, where numeric performance data tends 
to be prevalent, but there is also an aspect of self-awareness 
and community that can benefit from a connection enriched 
by carefully expressed data. In this context, through three de- 
sign workshops, we examine the sensory and semiotic quali- 
ty of the interactions, leveraging outputs in the form of haptic 
actuators, as well as subtle uses of lighting and kinematics. 
The exercises challenged participants to transform raw data 
into novel sensorial experiences through commonly available 
hardware (sensors and actuators) and software, constitut- 
ing a meta-medium to be valorized with the same agility as 
conventional materials, albeit with an obviously shifting set 
of skills. Based on the scientific literature and experimental 
results, this contribution offers a mapping of possible values 
between individual and shared experiences, working towards 
a new vocabulary of tactile and kinetic experiences that may 
be useful for the creative industries. 
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Introduction 

The boundaries between digital and physical reality are be- 
coming increasingly blurred, paving the way for new forms 
of hybrid interactions and opening up new meanings in per- 
ception. While extended Reality strives for a socially intercon- 
nected “metaverse” via VR and MR visors, this contribution 
tackles with interactions that often bypass the visual channel 
in order to connect people to (or through) data in more sen- 
sory ways, considering also haptic actuators (a growing com- 
munication channel, Zannoni et al., 2021). 

In many fields of application, including those outside of 
sports, tracking and monitoring technologies have entered the 
daily lives of people through various wearable devices for ob- 
serving the physical, mental, and emotional states of individu- 
als (Mukhopadhyay, 2015). This phenomenon of self-tracking 
via technological devices relates to the concept of the Quan- 


tified Self (QS) (Wolf et al, 2010), and can be considered a 
cultural movement dedicated to the collection of body data 
with the goal of assessing and improving one’s own physical, 
mental, and emotional performance. But going beyond the 
technical possibilities, the QS phenomenon has generated de- 
bate around its sociological and ethical implications in terms 
of privacy (Thuraisingham et al, 2018) and equality (Lupton, 
2015). In fact, while measuring specific body parameters of- 
fers objective data for evaluation and personal improvement, 
it also implies the continuous confrontation of a user's health 
and performance indicators with extraneous standards that 
may be above (or below) their realistic possibilities, leading to 
a potentially conflictual relationship between the QS and the 
sensitivity (Qualified Self?) of the users. 

In contrast to human behaviour, which is based on percep- 
tions and sensations, the data-driven approach to the self is 
capable of reconfiguring actions and responses that rely on 
objective measurements of data from devices, enacting a ma- 
nipulation that “could result in loss of human free agency and 
autonomy and even deceptions of humans" (Stephanidis et al., 
2019). As stated by Wiener (1950), technology has the power 
of influencing and changing human key values and behaviours, 
which is becoming increasingly evident in this era of rapid 
technological advancement and permeation in human socie- 
ty. This leads the research towards a more symbiotic relation- 
ship (Licklider, 1960) between devices and people, close to the 
concept of Human-Computer Confluence (Ferscha, 2016) and 
Human-Computer Integration (HCInt) (Farooq & Grudin, 2016; 
Mueller et al, 2020), leading to data-based communication 
through new forms of sensing, blurring the barriers between 
the physical body and the digital self. 


Computational Thinking and sensorial interactions 
This work is based on the assumption that Computational 
Thinking (the ability to define a step-by-step solution to com- 
plex problems; Wing, 2006) acquired through the practice 
of Learning by Doing (Bruce & Bloch, 2012) should be a part 
of every student's curriculum in order to better understand 
technology and develop a proactive approach to using tech- 
nology for improving everyday life. The relevance of Compu- 
tational Thinking (CT) is widely recognised and it is taught at 
various levels of general education or STEM-focused, from el- 
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ementary schools to universities (Sabitzer et al., 2014; Man- 
nila, 2014; Garcia-Pefialvo & Cruz-Benito, 2016). 

At the elementary school level, for example, an innovative 
education approach grounded in project-based teaching is 
proposed by Lyu (2021),tackling the children’s problem solv- 
ing skills while assisted by the teachers. Another didactic 
approach by Rossano (2020), consists of teaching middle 
school students the basics of Arduino and coding through 
simple exercises and applying the newly acquired knowledge 
to real-life situations, like the management of a wooden rail 
crossing model. At the high school level, Yin (2022) uses a 
didactic process with four steps to show how using Ardui- 
no-based activities can improve CT and engineering learning. 

On the level of higher education for the Design Profession, 
the ambitions of a CT-focused course can be more advanced, 
not only teaching but also inquiring about the possibility of 
turning digital, quantitative data into qualitative, sensorial 
feedback, and evaluating whether these can become an ac- 
tual framework. Beyond innovative functionalities, Design has 
a particular responsibility for finding appropriate semantic 
associations. New sensory qualities and gestures may follow 
analogue metaphors or pure abstraction, based on the intrinsic 
nature of the information being communicated or controlled. 
This paper presents the findings of a series of workshops, car- 
ried out with students of (industrial/product) design, with the 
goal of experimenting with such new sensory qualities. 

In particular, the general theme of sports has been chosen, 

as this provides wide opportunities for experimentation. Phys- 
ical activity is already one of the main targets of self-tracking, 
and sports are mostly quantitative, measurable, while also in- 
cluding various layers of social interactions between players, 
trainers, and fans. Indeed, past research works have shown 
interesting experimental augmentations of sporting activities, 
both within the boundaries of existing sports and reinventing 
them altogether. An interesting example by Hribernik (2022) 
developed a haptic feedback interface specifically for water 
sports, which pose obvious challenges, in order to effectively 
communicate data during a physical activity session. Anoth- 
er take on the haptic approach is tackled in the creation of a 
superhuman sport, “DubHap,” with the intention to redesign 
existing sports and adapt them to the new era (Chernyshov et 
al, 2018). With a design-driven approach, technology can also 
help to invent completely new “superhuman sports” for the 
AR/VR and sensing gloves, engaging citizens through a series 
of “ideathons” (participative design workshops) in order to pull 
in ideas, such as interacting with a virtual energy projectile that 
can only be sensed and not seen (Orikasa et al., 2017). 
Taking inspiration from the above mentioned research, our 
workshops’ participants started experimenting with isolated 
sensorial aspects of various sports, then proposed smart ob- 
jects that highlight a range of possibilities, analysed in the last 
section of the paper. 


Workshop structure 

The workshop format, called “Smart Design Pills”, has been 
developed for design students in the third BSc year, therefore 
with a solid background of creative exercises, modelling and 
presentation skills, but without any knowledge of interactive 
hardware & software prototyping or coding. The programme 
comprises three workshops that cover the fundamentals of 
prototyping, beginning with the simplest actuator and gradu- 
ally increasing difficulty: LEDs and light signals are the theme 


for the first workshop, “Shiny Rhythms”; vibrating signals and 
tactile perceptions are studied during the second workshop, 
“Touching Signals”; while movement and kinetics are ex- 
plored through the third workshop, “and Yet it Moves.” 

Each workshop is three weeks long and begins with an in- 
troductory exercise focused on computational thinking skills 
without digital hardware and software, by substituting and 
simulating the sensors/actuators with readily available tools 
and materials: the phone's flash and vibration motor are used 
for role playing, cardboard and clay for stop motion animation. 
Later during the first week, students learn about the sensors, 
actuators, and code necessary for the workshop, as well as 
studying relevant case studies of smart objects. The second 
step concentrates on defining a concept and developing a 
rough prototype of the group's interactive product in view of 
the third and final step, which is committed to creating a final 
prototype while also taking aesthetic factors into account. 

For the purposes of this course, students were divided into 
17 groups of two or three, and each was assigned a random 
sport. Numerous factors were considered in order to create a 
collection as diverse as possible, including the potential need 
for equipment, the degree of interaction (individual or team- 
based), and the settings in which the sports could be practised 
(gyms, particular outdoor settings, settings irrelevant to the 
practice). Based on their technical characteristics, the sports 
were divided into six macro-groups (water sports, balance 
sports, ball sports, racket sports, urban sports, and martial arts) 
to facilitate planning and review processes and the sharing 
of useful information between groups within the same mac- 
ro-group. 

Workshop 1 - “Shiny Rhythms,’ was based on the use of 
light signals via white LEDs. A couple of evocative images were 
selected for each sport, specifically drawing from visual, kinet- 
ic, and acoustic aspects, as inspiration or reference. The stu- 
dents were required to recreate each of these concepts using 
an animated light signal that included timing, rhythm, and in- 
tensity as variables, with the aid of particular apps that allowed 
them to control and modify the values. Sketches, notes, and 
illustrative graphs showing the intensity of the signal over time 
were required to document the brainstorming and design 
phases. Afterward, a brief video showcasing the three final per- 
formances had to be created. 

Students were then asked to apply the knowledge they had 
acquired to a product that would incorporate the use of LEDs 
into their assigned sports activities as the actual exercise for 
the first workshop, proceeding in the three weeks process as 
described above. The final goal was to elaborate digital signals 
into a light performance that could visually express the quali- 
ties of the sport. 

Workshop 2 - “Touching Signals”, focused on experiment- 
ing with the perception of haptic signals using vibrating actua- 
tors. The materials provided for the introductive exercise were 
an illustration of the emotion spectrum (Plutchik, Robert, 1980) 
as well as a map of emotions perceived by the human body 
(Nummenmaa et al., 2013) that helped them identify the areas 
of the body where sensors and actuators should be placed. The 
introductory activity focused on the representation of a des- 
ignated emotion through vibrating sequences. The purpose of 
the activity was to find haptic feedback for conveying infor- 
mation that is typically processed through visual or auditory 
channels during the training processes, athletic performance, 
and communication with other participants (fans, teammates, 
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coach). The actual three weeks long activity of this workshop 
requested the students to design a product that incorporated 
haptic actuators in sports activities. 

Workshop 3 - “And Yet it Moves", requested to design a ki- 
netic smart object capable of interacting with the athletes, the 
audience, or the sporting environment. To familiarise students 
with the concept of movement, they were asked to identify 
some key aspects of sport (e.g, strength, coordination, ac- 
celeration) and represent them through a stop-motion video, 
using clay. Subsequently, they were asked to focus on three 
categories (objects for training, for sport execution, and for 
communication), elaborating a concept for each situation, 
rough prototyping and physically simulating the interaction of 
their products. Based on the findings from this exercise, stu- 
dents designed the project for the third workshop. 


Figure 1. Top row: a storyboard for each project; bottom row: 
the prototypes resulted from the workshops. 
Designed by: Barberis Lorenzo, Ciubotaru Gabriel Cosmin, Guarnieri Elena. 


Results can be illustrated through the projects of the table 
tennis group (Flgure 1). The interactive table tennis racket 
designed for the first workshop features built-in LED lights 
that react to the ball’s stroke through a shock sensor and the 
racket’s movement through an accelerometer, providing a 
visually appealing indicator for the most spectacular (quick- 
est, strongest) moves during the game. The product from 
the second workshop is a wearable harness for the player, 
equipped with vibration actuators on each shoulder. Each 
vibrating actuator responds to the audience's cheers on its 
respective side, giving the player haptic feedback on their 
surroundings without distraction from the game. Training 
the player with a small wheeled robot is the aim of the final 
movement-themed project. The robot is designed to run in 
random directions and it must be hit with the ball to raise the 
speed and therefore difficulty. 


Interactive approaches 

The 51 interactive projects developed across the three work- 
shops lend themselves to a categorisation that can identify 
various aspects along which the projects are diversified. Pri- 
or to analysing our case studies, it’s worth noting that past 
research has examined and theorised on qualitative design 
values in a variety of ways. Some of the methodologies that 
emerged from the research have been applied to the projects 
as a first step to approaching a more fitting system of values 
for this subject. Among the pioneers of Design for loT, Rose 


(2014) categorised projects according to 7 possible char- 
acteristics: Glanceability (439 of the 51 projects fall under 
this characteristic), Gestureability (4196), Affordability (5996), 
Wearability (4796), Indestructibility (4396), Usability (5696), 
Loveability (33%). According to Rose, designing a product 
that exhibits all seven characteristics is quite difficult. In fact, 
only 10% of the 51 projects displayed one or two of the traits; 
around 30% displayed three or four; barely 15% displayed five 
out of seven; and only 4.5% presented six out of the seven 
characteristics. On the other hand, a more generic categori- 
zation such as the well-known “three levels of design” (Nor- 
man, 2004) - Reflective, Behavioural and Visceral - may be 
(even more) useful to better understand the product. 

Beyond pre-existing categories, the traits of each student 
project were analysed separately, thus identifying a set of con- 
notation parameters that were common to all of the projects. 
Each parameter presents two dualistic and mutually exclusive 
values: one that leans on a “personal” and “closed” aspect and 
one on a “shared” and “open” side. The parameters that have 
been identified are as follows, naming the “personal” value first 
and the “shared” value second: 

P.1 - Instinctive / New language: Products marked “instinc- 
tive” appeal to the immediacy of human instincts and com- 
mon knowledge, whereas products marked “new language” 
present a new vocabulary of symbols that the user must learn 
in order to interact with the device. An example of an instinc- 
tive project is a parkour-specific leg band that vibrates in the 
event of a violent landing, simulating the wave of impact from 
the ground upwards. A rotating and portable basketball hoop 
that can be attached to any vertical pole is a project that intro- 
duces a new language, revolutionising the traditional game of 
basketball. 

P.2 - Individual / Collective: Products marked “Individual” 
are defined by a single user, whereas “collective” projects rely 
on the participation of multiple users who contribute to the 
device's functioning. The climbing belt is an example of an 
individual product that, through vibration, “pushes” and ac- 
companies the climber towards the wall, correcting the po- 
sition without the help of instructors. The floats that confine 
the water polo field demonstrate the collective aspect. These 
light up to show the intensity of the fans’ cheers for the play- 
ers in the water. 

P.3 - Performance / Entertainment: The projects that pres- 
ent the value of “performance” are those focused on pure 
sports execution (competitive or training). Instead, “entertain- 
ment" indicates the projects that highlight the visual or spec- 
tacular aspect for those who are watching. The fencing target 
that moves to dodge the shots of the fencer is an example 
of a product for sports performance and training. The kinetic 
sculpture in the centre of the skateboard arena, which reacts 
to the noise of the fans, effectively expresses the concept of 
entertainment. 

P.4 - Human-driven/ Device-driven: Products marked “Hu- 
man-driven” react to user input, detecting human interac- 
tion and then providing feedback. On the other hand, the “de- 
vice-driven” products are the first to provide an indication to 
the users to guide them through the practice. A bracelet de- 
signed for boxing fans is a “human-driven” product that pro- 
vides vibrant feedback to the wearer based on the power of the 
boxer’s blows. Whereas, a small wheeled robot equipped with 
a chalk that traces a random path on the floor for the skater to 
follow is an example of a “device-driven” approach. 
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P.5 - Wearable/ Tangible: These two values indicate the 
product's location in relation to the user: “Wearable” projects 
are products that are actually worn by the user and remain in 
direct contact with the body; “tangible” products are physi- 
cal devices that can be either individual objects or applied to 
sports equipment. An example of a “wearable” device is com- 
posed of two leg bands designed for paddle tennis that vibrate 
alternately to indicate to the player the correct rebound to 
keep during the game. A “tangible” device applied to the hock- 
ey stick warns the player by vibrating when the disc moves too 
far, allowing them to maintain control. 

The visualisation of the data (Figure 2) has made it evi- 
dent that the majority of the products that resulted from the 
course tend to place in the “personal” sphere (659) rather than 
the “shared” sphere (35%). In fact, the majority of the projects 
displayed characteristics that were considered personal (in- 
stinctive, individual, performative, human-driven, wearable). 
The third workshop marked a high point for wearable product 
design, which is one of the main causes of this result. This out- 
come demonstrates that the interaction between humans and 
devices is mostly viewed as a private experience, but on the 
other hand, the tendency to involve the collectivity is relevant. 
Perhaps as human-machine contact advances, it will become 
more diverse and inclusive to the point where it serves as a tool 
for strengthening human unity. 
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Figure 2. Result of the systematisation of projects through 
the “personal” and “shared” parameters. 


Conclusions 

The research conducted to date has made it possible to test 
the “Smart Design Pills” didactic format with the goal of re- 
fining and verifying it in the future also in other fields. The 
programme includes the formulation of three workshops 
that focus on the human-device symbiosis, in this specif- 
ic case in the field of sports, with the aim of transforming 
quantitative digital signals into sensory and qualitative inter- 
actions. Each workshop focuses on a fundamental concept 
and is approached through introductory activities giving the 
participants an alternative perspective before tackling the 
actual project. The products obtained from the workshops 
confirmed the ability of the participants to start from a basic 
level of interactive prototyping, but still achieve meaningful 
results, developing significantly their skill of computational 
thinking in the meantime. Five versatile parameters with po- 
larised values that lean either to a “personal” sphere or to a 
“social” sphere have been developed based on the evaluation 
of the projects that resulted from the course. The outcome 
of this distinction shows a propensity to view human-ma- 
chine connection as an individual experience, however the 
encouraging quantity of collective interactions may provide 
an opening to a more expansive definition of “human-de- 
vice” interaction. 

From the perspective of the Cumulus community, it would 
be interesting to see whether and how such principles can be 
applied by students (and professionals) of different cultural 
backgrounds (compared to Rome, Italy) and in different fields 
of activity. Finally, the research confirms the value of Quanti- 
fied Self as a tool to “reflect, learn, remember and improve” 
(Wolf, 2010) on the individual's performance, but also confirms 
a possible “qualified self” approach by treating raw numerical 
data as a design material, that can lead to engaging experi- 
ences and ultimately a more a symbiotic experience between 
people and data. 
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